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(54) Music reproducing system, rhythm analyzing method and storage medium 



(57) In a music reproducing system, each of charac- 
ters is allocated in advance a favorite rhythm compo- 
nent and a peculiar figure expressing ability A sound 
pressure data producing section produces sound pres- 
sure data for each of a plurality of frequency bands from 
musical data. A frequency band identifying section iden- 
tifies the frequency band which beats a rhythm most. A 
rhythm estimating section derives a changing period of 
a sound pressure component in the sound pressure 
data at the identified frequency band and estimates a 
rhythm component based on the derived changing 
period. A character managing section cumulatively 
changes a figure expressing ability of a displayed char- 
acter according to the matching degree between the 
estimated rhythm component and a favorite rhythm 
component of the displayed character. A display control 
section changes a display figure of the displayed char- 
acter according to the figure expressing ability thereof 
while musical data is reproduced. 
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Description 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

[0001] The present invention relates to a music 
reproducing system having a function of accurately 
extracting a rhythm component of musical data upon 
reproduction thereof and changing a display figure of a 
character based on the extracted rhythm component. 

DESCRIPTION OF THE RELATED ART 

[0002] There are many personal computers and 
video game apparatuses provided with a device for 
reading a CD-ROM storing program codes and various 
data. Most of them can read a music CD storing musical 
data and reproduce music from the musical data. 
[0003] Further, there have recently appeared video 
game apparatuses with a music reproducing function, 
wherein a character in a CG (computer graphic) image 
is displayed on a display screen and dances autono- 
mously or via an external operation to the accompani- 
ment of a rhythm of reproduced music, so that a game 
player can enjoy not only the reproduced music itself, 
but also visually. 

[0004] For example. JP-A-1 0-99546 introduces a 
video dance game apparatus, wherein at least two 
dance players, i.e. characters, are displayed on a dis- 
play screen, and a dance contest is carded out depend- 
ing on a time difference between the timing at which 
dance technique data of each character is read out via 
an external operation and the timing at which beat infor- 
mation (information forming the basis of a rhythm of 
music) contained in dance music is produced. This 
video dance game apparatus can use only dance music 
as music (musical data) for extracting the beat there- 
from. 

[0005] In the foregoing video game apparatuses, 
beat extraction is implemented using a filter on the 
assumption that reproduced music is dance music so 
that the frequencies of an instrument beating a rhythm, 
i.e. the beat generation frequencies, are concentrated to 
a low frequency band, such as a drum sound. However, 
there is such dance music wherein the beat generation 
frequencies do not necessarily fall within the low fre- 
quency band. In this event, the accurate beat can not be 
extracted, thus resulting in lack of objectivity of a dance 
contest. 

[0006] Further, in the conventional technique, it is 
not possible to accurately analyze the rhythm of music, 
other than dance music, reproduced from a commercial 
music CD wherein a variety of instruments beat the 
rhythm, for reflection upon a display figure of a charac- 
ter. 



SUMMARY OF THE INVENTION 

[0007] Therefore, it is an object of the present 
invention to provide a music reproducing system which 

5 can accurately reflect a rhythm component contained in 
musical data upon a display f igure of a character. 
[0008] It is another object of the present invention to 
provide a rhythm analyzing method which can accu- 
rately analyze a rhythm component of musical data. 

10 regardless of a genre such as dance music. 

[0009] It is another object of the present invention to 
provide a storage medium for realizing the foregoing 
music reproducing system on a computer. 
[0010] According to one aspect of the present 

15 invention, there is provided a music reproducing system 
wherein a character is allocated in advance a favorite 
rhythm component and a display figure of the character 
on a screen Is changed based on musical data, the sys- 
tem comprising frequency band identifying means for 

20 producing sound pressure data for each of a plurality of 
frequency bands from the musical data and for identify- 
ing, based on the sound pressure data for the plurality 
of frequency bands, one of the plurality of frequency 
bands which beats a rhythm most; rhythm estimating 

25 means for deriving a changing period of a sound pres- 
sure component from the sound pressure data for the 
identified frequency band and for estimating a rhythm 
component of the sound pressure data for the identified 
frequency band based on the derived changing period; 

30 and control means for changing the display figure of the 
character according to a matching degree between the 
estimated rhythm component and the favorite rhythm 
component. 

[0011] It may be arranged that the musical data is 
35 Stored in a portable storage medium and the frequency 
band identifying means is activated when the musical 
data is read into the music reprodudng system from the 
portable storage medium. 

[0012] It may be arranged that the rhythm estimat- 

40 ing means derives the changing period by applying an 
FFT process to the sound pressure data for the Identi- 
fied frequency band, superimposes auxiliary data hav- 
ing a period integer times the changing period onto the 
sound pressure data for the identified frequency band 

45 SO as to suppress components other than the rhythm 
component, and applies an Inverse FFT process to the 
sound pressure data superimposed with the auxiliary 
data so as to estimate the rhythm component. 
[0013] It may be arranged that the rhythm estimat- 

50 ing means verifies whether the changing period is cor- 
rect and, when the changing period Is verified to be 
correct, the rhythm estimating means outputs the 
rhythm component for a predetermined time. 
[0014] According to another aspect of the present : 

55 invention, there is provided a music reproducing system 
connected to a display unit and provided with music 
inputting means and music reproducing means, the sys- 
tem comprising first storing means for storing a favorite 
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rhythm component for each of characters; second stor- 
ing means for storing a figure expressing ability for each 
of the characters, the figure expressing ability used for 
expressing a display figure of the corresponding charac- 
ter to be displayed on a screen of the display unit; fre- 
quency band identifying means for producing sound 
pressure data for each of a plurality of frequency bands 
from musical data inputted via the music Inputting 
means, and for identifying, based on the sound pres- 
sure data for the plurality of frequency bands, one of the 
plurality of frequency bands which beats a rhythm most; 
rhythm estimating means for deriving a changing period 
of a sound pressure component from the sound pres- 
sure data for the identified frequency band and for esti- 
mating a rhythm component of the sound pressure data 
for the identified frequency band based on the derived 
changing period; and control means for causing one of 
the characters to be displayed on the screen of the dis- 
play unit, reading the favorite rhythm component of the 
one of the characters from the first storing means, 
updating the figure expressing ability of the one of the 
characters stored in the second storing means accord- 
ing to a matching degree between the favorite rhythm 
component read from the first storing means and the 
rhythm component estimated by the rhythm estimating 
means, and changing the display figure of the displayed 
character based on the updated figure expressing abil- 
ity to the accompaniment of the musical data repro- 
duced by the music reproducing means. 
[0015] It may be arranged that the control means 
updates the figure expressing ability of the displayed 
character so as to be cumulatively increased every time 
musical data containing the favorite rhythm component 
is reproduced. 

[001 6] It may be arranged that the music reproduc- 
ing system further comprises background image control 
means for changing a background image of the dis- 
played character according to the figure expressing abil- 
ity- 

[0017] It may be arranged that the second storing 
means is an exchangeable nonvolatile storage medium. 
[0018] According to another aspect of the present 
invention, there is provided a rhythm analyzing method 
to be implemented in an apparatus which displays a 
character allocated in advance a favorite rhythm compo- 
nent oh a screen based on a rhythm component of 
musical data reproduced, the method comprising the 
steps of producing sound pressure data for each of a 
plurality of frequency bands from the musical data, and 
identifying, based on the sound pressure data for the 
plurality of frequency bands, one of the plurality of fre- 
quency bands which beats a rhythm most; deriving a 
changing period of a sound pressure component from 
the sound pressure data for the identified frequency 
band, and estimating a rhythm component of the sound 
pressure data for the identified frequency band based 
on the derived changing period; and reflecting a match- 
ing degree between the estimated rhythm component 



and the favorite rhythm component onto control infor- 
mation for changing a display figure of the displayed 
character. 

[0019] It may be arranged that the step of deriving 

5 the changing period and estimating the rhythm compo- 
nent comprises deriving the changing period by apply- 
ing an FFT process to the sound pressure data for the 
identified frequency band, superimposing auxiliary data 
having a period integer times the changing period onto 

10 the sound pressure data for the identified frequency 
band so as to suppress components other than the 
rhythm component, applying an inverse FFT process to 
the sound pressure data superimposed with the auxil- 
iary data and performing a verification process for the 

IS sound pressure data applied with the inverse FFT proc- 
ess so as to estimate the rhythm component 
[0020] It may be ananged that the rhythm analyzing 
method further comprises outputting the estimated 
rhythm component for a predetermined time. 

20 [0021] According to another aspect of the present 
Invention, there is provided a storage medium storing 
program codes readable by a computer which displays 
a character allocated in advance a favorite rhythm com- 
ponent on a screen based on a rhythm component of 

25 musical data, the program codes causing the computer 
to execute the steps of producing sound pressure data 
for each of a plurality of frequency bands from the musi- 
cal data, and identifying, based on the sound pressure 
data for the plurality of frequency bands, one of the plu- 

30 rality of frequency bands which beats a rhythm most; 
deriving a changing period o1 a sound pressure compo- 
nent from the sound pressure data for the identified fre- 
quency band, and estimating a rhythm component of 
the sound pressure data for the identified frequency 

35 band based on the derived changing period; and reflect- 
ing a matching degree between the estimated rhythm 
component and the favorite rhythm component onto 
control information for changing a display figure of the 
displayed character. 

40 [0022] It may be ananged that the program codes 
cause an operating system of the computer or other 
program codes to execute a part of the steps. 
[0023] It may be arranged that the storage medium 
also stores the musical data and is separable from the 

45 computer. 

[0024] According to another aspect of the present 
invention, there is provided a storage medium storing 
program codes readable by a computer which displays 
a character allocated In advance a favorite rhythm com- 

50 ponent on a screen based on a figure expressing ability 
peculiar to the character and a rhythm component of 
musical data reproduced, the program codes causing 
the computer to execute the steps of producing sound 
pressure data for each of a plurality of frequency bands 

55 from the musical data, and identifying, based on the 
sound pressure data for the plurality of frequency 
bands, one of the plurality of frequency bands which 
beats a rhythm most; deriving a changing period of a 
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sound pressure component from the sound pressure 
data tor the identHied frequency band, and estimating a 
rhythm component of the sound pressure data for the 
identified frequency band based on the derived chang- 
ing period; and updating the figure expressing ability 
according to a matching degree between the estimated 
rhythm component and the favorite rhythm component. 
[0025] It may be arranged that the program codes 
cause the computer to execute a step of changing a 
background image of the character according to the fig- 
ure expressing ability. 

[0026] It may be arranged that the program codes 
cause an operating system of the computer or other 
program codes to execute a part of the steps. 
[0027] It may be arranged that the storage medium 
also stores the musical data and is separable from the 
computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0028] The present invention will be understood 
more fully from the detailed description given hereirtoe- 
low. taken in conjunction with the acconpanying draw- 
ings. 

[0029] In the drawings: 

Fig. 1 is a block diagram showing a hardware struc- 
ture of a video game apparatus for realizing a music 
reproducing system according to a preferred 
embodiment of the present invention: 
Fig. 2 is a functional block diagram of the music 
reproducing system according to the preferred 
embodiment of the present invention; 
Fig. 3 Is a flowchart showing the overall operation 
sequence of the music reproducing system shown 
in Fig. 2; 

Fig, 4 is a flowchart of a subroutine for rhythm com- 
ponent extraction included in the flowchart shown in 
Fig. 3; 

Fig. 5 Is a diagram for explaining a technique of 
rhythm component extraction according to the pre- 
ferred embodiment of the present invention: 
Fig. 6 is a diagram showing an example of a dis- 
played image based on a low figure expressing abil- 
ity according to the preferred embodiment of the 
present invention; 

Fig. 7 is a diagram showing an example of a dis- 
played image based on an increased figure 
expressing ability according to the preferred 
emlXKJiment of the present invention; and 
Fig. 8 is a diagram showing an example of a dis- 
played image based on a further increased figure 
expressing ability according to the preferred 
embodiment of the present invention, 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0030] Now. a music reproducing system according 



to a preferred embodiment of the present invention will 
be desaibed with reference to the accompanying draw- 
ings. In this embodiment, the music reproducing system 
is realized using a known video game apparatus being 
5 a kind of a computer, but may also be realized using a 
general purpose personal computer having a display 
unit. 

[0031] In this embodiment, each of characters and 
a background thereof to be displayed are CG images in 

10 the form of polygons, and a display figure, such as a 
motion of each character and an expression thereof, 
and a variation of the display figure are realized by con- 
trol information including program codes for changing 
the shape of the polygons, and a later-described rhythm 

75 analysis result and other variation parameters, and fur- 
ther including a figure expressing ability peculiar to each 
character and background data. 

[0032] The variation parameters except the figure 
expressing abilities of characters, program codes for 

20 realizing the music reproducing system on the compu- 
ter, the program codes for changing the shape of the 
polygons and the background data (hereinafter collec- 
tively referred to as "game program") are prestored in a 
CD-ROM as an example of a storage medium. Musical 

25 data is stored in a commercial music CD separate from 
the foregoing CD-ROM. The figure expressing abilities 
are stored in another storage medium described later. 
[0033] In this embodiment, "musical data" repre- 
sents data indicative of characteristics of instrument 

30 tones, voices and other tones, "figure expressing ability" 
represents a parameter for expressing magnitudes of 
motions of a corresponding character and affluence of 
an expression thereof. The figure expressing ability is 
allocated per character and the value thereof is cumula- 

35 tively changed (increased or decreased) depending on 
reproduced music, "background data" is a parameter for 
changing a background of a corresponding character 
and is associated with the figure expressing ability. Spe- 
cifically, different background data is selected according 

40 to the value of the figure expressing ability. 

[Video Game Apparatus] 

[0034] Now, the video game apparatus for constitut- 
45 ing the music reproducing system will be described. For 
the video game apparatus, the technique described in, 
for example, JP-A-8-212377 can be used. 
[0035] Specifically, in this embodiment, the forego- 
ing video game apparatus is used for reading the game 
50 program stored in the foregoing CD-ROM which is 
exchangeable and portable. By executing the game pro- 
gram independently or executing it cooperatively with an 
operating system (OS) of the video game apparatus 
and other program codes in the apparatus, the music 
55 reproducing system is constructed to implement repro- 
duction of music, production and display of CG images, 
an analysis of the rhythm component contained in musi- 
cal data, a display control of the CG images based on a 
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result of the rhythm analysis. 

[0036] As shown in Fig. 1 . the video game appara- 
tus 1 comprises a main control section 10. an image 
processing section 20. a sound processing section 30, a 
disk control section 40. a communication control section 
50. a parallel input/output (I/O) port 61, a serial 
input/output (I/O) port 62 and a main txjs B connecting 
the foregoing components 10-50. 61 and 62 to allow 
two-way communications among them, 
[0037] The main control section 10 comprises a 
CPU n. a peripheral device controller 12 tor executing 
an interrupt control, a DMA (direct memory access) 
transfer control, etc.. a main memory 13 for storing the 
game program etc.. and a ROM 14 storing an operating 
system (OS) etc. for performing a general control of the 
components in the apparatus. The CPU 11 is a RISC 
(reduced instruction set computer) CPU, which exe- 
cutes the OS stored in the ROM 14 and the game pro- 
gram stored in the main memory 1 3 to realize a plurality 
of functional blocks which will be described later. 
[0038] The image processing section 20 comprises 
a geometry transfer engine (GTE) 21 carrying out high- 
speed coordinate transformation relative to data to be 
displayed, a graphics processing unit (GPU) 22 for per- 
forming a drawing process of CG images (character, 
background thereof, etc.) composed of combinations of 
polygons, such as triangles and quadrilaterals, and 
sprites according to drawing commands from the CPU 
11,8 frame buffer 23 for temporarily storing the CG 
images produced by the drawing process at the GPU 
22. and an image decoder (MDEC) 24 for decoding 
image data as requested. The display unit 25 reads the 
CG images from the frame buffer 23 and displays them. 
[0039] By consecutively executing the drawing 
process of CG images by the GPU 22 and the storing of 
them into the frame buffer 23. the CG images including 
motion elements, such as a variation in motion and 
expression of a character, can be displayed on the dis- 
play unit 25. 

[0040] The sound processing section 30 comprises 
a sound processing unit (SPU) 31 for reproducing music 
based on musical data, a sound tnjffer 32 for tenrporar- 
ily storing the musical data, and a loudspeaker 33. 
[0041 ] The disk control section 40 comprises a disk 
drive 41 for reproducing the stored contents of the game 
program CD-ROM 44 or the music CD so as to be rec- 
ognized by the CPU 11 (hereinafter referred to as 
"reproduced data"), a CD-ROM decoder 42 for decod- 
ing the reproduced data when error correcting codes 
(ECC) are added thereto, and a CD-ROM buffer 43 for 
temporarily storing the reproduced data from the disk 
drive 41 before storing it into the main memory 13. 
[0042] The CD-ROM decoder 42 also constitutes a 
part of the sound processing section 30. /^ong outputs 
of the CD-ROM decoder 42. outputs relating to musical 
data are inputted into the SPU 31 
[0043] The communication control section 50 com- 
prises a communication control device 51 for controlling 



communication with the CPU 11 via the main bus B, a 
controller 52 for inputting commands from a game 
player, and an exchangeable memory card 53 as an 
exannple of an nonvolatile storage medium. 

5 [0044] The controller 52 is an Interface for inputting 
the will of the game player, such as commands for 
selecting a character, inputting musical data and repro- 
ducing music. The command is fed to the communica- 
tion control device 51 via synchronous communication, 

10 [0045] The communication control device 51 sends 
command data representing the command from the 
controller 52 to the CPU 1 1. In this fashion, the will of 
the game player is notified to the CPU 1 1 so that the 
CPU 1 1 executes a process corresponding to the will of 

75 the game player based on the executing game program. 
The communication control device 51 also has a func- 
tion of writing and reading data into and from the mem- 
ory card 53 based on a command from the CPU 11. 
Since the memory card 53 is separated from the main 

20 bus B, it can be loaded or unloaded with the power 
being on. Accordingly, a figure expressing ability per 
character and various kinds of set data can be stored in 
a memory card 53 peculiar to a game player. 
[0046] A sound processing apparatus, such as an 

25 audio apparatus or a singing play unit system, for repro- 
ducing nrBJSic may be connected to the parallel 
Input/output (I/O) port 61 or the serial input/output (I/O) 
port 62 so that music from the sound processing appa- 
ratus can be inputted via the port 61 or 62. Accordingly. 

30 the present invention can be carried out using musical 
data inputted from not only a music CD but also an 
external sound processing apparatus. 



35 



[Music Reproducing System] 



[0047] When the foregoing video game apparatus 1 
is turned on or reset while the CD-ROM 44 and the 
memory card 53 being loaded, the CPU 1 1 executes the 
OS stored in the ROM 14 to first initialize the whole 
40 apparatus, then control the disk control section 40 so as 
to read the game program from the CD-ROM 44 into the 
main memory 13. 

[0048] After the completion of the reading, the CPU 
1 1 gives a guidance to load the music CD instead of the 

45 CD-ROM 44. Then, following the guidance, the game 
player loads the music CD into the disk drive 41 . 
[0049] Through execution of the foregoing game 
program (including cooperative execution thereof with 
the OS and other program codes), the CPU 11 pro- 

50 duces the functional blocks as shown in Fig. 2 so that 
the music reproducing system generally designated by 
numeral 2 is realized. 

[0050] The music reproducing system 2 comprises 
a sound pressure data producing section 211. a fre- 
55 quency band identifying section 212. a rhythm estimat- 
ing section 213, a character managing section 214. a 
character data file 215. a character image processing 
section 216, a background image processing section 



NSDOClD: <EP 1031363A2J,> 



9 



EP 1 031 363 A2 



10 



217 and a display control section 218. 
[0051] The sound pressure data producing section 
21 1 produces sound pressure data for each of a plural- 
ity of frequency bands from inputted musical data (base 
data). The frequency band identifying section 212 iden- 
tifies the frequency band which beats a rhythm most, 
based on the produced sound pressure data for the plu- 
rality of frequency bands. The rhythm estimating section 

213 derives a changing period of a sound pressure 
component from the sound pressure data of the identi- 
fied frequency band and estimates a rhythm component 
of the sound pressure data of the identified frequency 
band based on the derived changing period. The 
detailed processing will be described later. 

[0052] The character managing section 214 man- 
ages display data relating to a plurality of characters 
and their backgrounds stored in the character data file 
215 and the memory card 53. For example, the charac- 
ter managing section 214 reads a character figure form- 
ing element from the character data file 215 based on a 
command given by the game player via the controller 
52, reads a peculiar figure expressing ability stored per 
character in the memory card 53, judges the matching 
degree between "favorite rhythm component" allocated 
per character in the character data file 215 and a rhythm 
component extracted or estimated from musical data, 
updates a figure expressing ability of the corresponding 
character based on a result of the judgment, and reads 
background data from the character data file 215. 
[0053] Further, when the game player inputs identi- 
fication data for identifying one of the characters, the 
character managing section 214 sends a figure forming 
element corresponding to a figure expressing ability of 
the identified character on that time point, to the charac- 
ter image processing section 216. In addition, every 
time the figure expressing ability reaches a predeter- 
mined reference value, the character managing section 

214 sends predetermined command data to the back- 
ground image processing section 217. 

[0054] lavorite rhythm component" stored in the 
character data file 215 represents a rhythm band univo- 
cally predetermined per character for ensuring intrinsic 
motions and expressions of the corresponding charac- 
ter and is a parameter to be used as an object for judg- 
ing the matching degree relative to a rhythm component 
contained in reproduced music, "favorite rhythm compo- 
nent" is allocated per character upon system design or 
upon initial setting by a game player. 
[0055] In this embodiment, each character is allo- 
cated, as a favorite rhythm component, one of a first 
rhythm band where one beat is 10/30 [sec] to 13/30 
[sec], a second rhythm band where one beat is 14/30 
[sec] to 1 7/30 [sec], and a third rhythm band where one 
beat is 18/30 [sec] to 24/30 [sec]. 
[0056] The character image processing section 216 
implements image processing relating to character fig- 
ure display, the background image processing section 
217 implements image processing relating to character 



background display, and the display control section 218 
implements a display control for characters and their 
backgrounds. 

[0057] Now. an operation of the music reproducing 
5 system 2 will be described. 

[0058] Referring to Fig. 3. the overall operation 
sequence of the music reproducing system 2 will be first 
described. 

[0059] When identification data of a character and a 

10 background selected by a game player is inputted via 
the controller 52 (step SlOl). the character managing 
section 214 causes the character and the background 
to be displayed on the display unit 25 via the character 
image processing section 216. the background image 

15 processing section 217 and the display control section 
218 (Step SI 02). Further, the character managing sec- 
tion 214 reads a figure expressing ability and other set 
data with respect to the displayed character from the 
memory card 53. 

20 [0060] When music is reproduced based on musi- 
cal data stored in the music CD loaded in the disk drive 
41 (step 3103), a rhythm component of the reproduced 
music is extracted (step Si 04). The rhythm component 
extraction technique will be described later in detail. 

25 [0061] The character managing section 214 judges 
to what degree the extracted rhythm component 
matches or agrees with a favorite rhythm component of 
the displayed character. For brevity of description, it is 
assumed here that the character managing section 214 

30 only judges whether or not a frequency band of the 
extracted rhythm component agrees with a frequency 
band of the fevorite rhythm component (i.e. one of the 
foregoing first to third rhythm t>ands). If the frequency 
band of the extracted rhythm component is other than 

35 any of the first to third rhythm bands, a rounding process 
is carried out to round the extracted rhythm component 
irrto the most approximate rhythm band. 
[0062] If both rhythm components agree with each 
other, the character managing section 214 updates the 

40 figure expressing ability of the displayed character so as 
to be increased (Yes at step 8105; step 8106). On the 
other hand, if both do not agree with each other, the 
character managing section 214 updates the figure 
expressing ability of the displayed character so as to be 

45 lowered (No at step S105; step 3107). Then, the char- 
acter managing section 214 changes in real time the 
display figure of the displayed character and (back- 
ground according to the updated figure expressing abil- 
ity (step S108). The timing of changing the display 

so figure is matched with the extracted rhythm component, 
i.e. the display figure is changed to the accompaniment 
of the extracted rhythm component. 
[0063] As the number of times of agreement of an 
extracted rhythm component with a favorite rhythm com- 

55 ponent of a displayed character increases, i.e. as the 
number of times of reproduction of music including a 
favorite rhythm component of a displayed character 
increases, a figure expressing ability of the displayed 
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Character is updated so as to be cumulatively 
increased. The character managing section 214 moni- 
tors a change of the figure expressing ability and. every 
time a value of the figure expressing ability reaches a 
reference value, the character managing section 214 
displays a predetermined effect image on the back- 
ground of the character (Yes at step S109; step S1 10). 
As an effect image, it may be arranged, for example, to 
flash the background to the accompaniment of the 
rhythm component or to separately display a special 
item or character. 

[0064] During reproduction of music, execution of 
step S104 and subsequent steps is repeated (No at 
step S1 11). When the music reproduction is finished, 
the figure expressing at»lity and other set data of the 
displayed character are stored (updated) into the mem- 
ory card 53, and the operation sequence is finished (Yes 
at step S1 11). 

( Rhythm Component Extraction Process) 

[0065] Now. the rhythm component extraction proc- 
ess executed at step 8104 will be described in detail 
with reference to Figs. 4 and 5, 

[0066] The rhythm component can be considered 
as a sound pressure wave of a constant period. Further, 
in case of musical data, the rhythm component is nor- 
mally produced by a particular Instrument which is 
determined according to a genre. Moreover, there are 
mixed various instrument sound data in musical data, 
including those beating a constant rhythm and those not 
beating a constant rhythm. 

[0067] Accordingly, in the sound pressure data pro- 
ducing section 211, an FFT (Fast Fourier Transforma- 
tion) process is applied to musical data (t^ase data) of 
several frames shown at (a) in Fig. 5 so as to produce 
sound pressure data at frequency bands A, B and C as 
shown at (b). (c) and (d) in Fig. 5. respectively (step 
S201). One frame is set to. for example. 1/30 sec. and 
the FFT process is performed at 44.1 [kHz] for 1/30 
[sec]. The reason why the sound pressure data for the 
plurality of frequency bands are produced is that instru- 
ments have their own sound characteristics, for exam- 
ple, a bass drum sound is concentrated to low 
frequencies, while a drum sound is difficult to pick up at 
200 [Hz]. 

[0068] From the sound pressure data shown at (b) 
to (d) in Fig. 5, it is still not possible to judge which fre- 
quency band (instrument) beats a rhythm. Accordingly, 
the sound pressure data producing section 21 1 further 
applies an FFT process to the sound pressure data at 
the frequency bands A, B and C shown at (b) to (d) In 
Fig. 5, so as to obtain sound pressure data as shown at 
(e). (f) and (g) in Fig. 5. 

[0069] Then, from the sound pressure data shown 
at (e) to (g) in Fig. 5, the frequency band identifying sec- 
tion 212 identifies the frequency band which beats a 
rhythm most, i.e. identifies the frequency band B in Fig. 



5(stepS202). 

[0070] Specif ically, if the sound pressure data has a 
constant period, a given large value appears at a certain 
frequency after applying the foregoing second FFT 
5 process (i.e. in the sound pressure data shown at (e) to 

(g) in Fig. 5). The frequency band identifying section 
212 judges whether there exists such a large value in 
the sound pressure data shown at (e) to (g) in Fig, 5. If 
judging positive, the frequency band identifying section 

10 212 determines that the frequency band (frequency 
band B in Fig. 5) corresponding to the sound pressure 
data Including such a large value (sound pressure data 
shown at (f) in Fig. 5) is a frequency band which beats a 
rhythm most. 

15 [0071] After identifying the frequency band (instru- 
ment) beating the rhytiim most, the rhyttim estimating 
section 213 derives a changing period of a sound pres- 
sure component (amplitude component) from the corre- 
sponding sound pressure data (shown at (f) in Fig. 5) 

20 (step S203). Then, the rhythm estimating section 213 
superimposes auxiliary data havirig a period integer 
times the derived changing period onto the sound pres- 
sure data (shown at (c) in Fig. 5). so as to suppress 
components other than the rhythm beating component 

25 to a value of 0 (zero), and then applies an inverse FFT 
process thereto (step S204). Fig. 5 at (h) shows sound 
pressure data obtained as a result of applying the 
inverse FFT process. 

[0072] Thereafter, for enhancing ttie rhythm extrac- 
30 tlon accuracy, the rhythm estimating section 213 verifies 
or inspects whether the sound pressure data (shown at 

(h) in Fig. 5) after the application of ttie inverse FFT 
process is sampled at the expected period (derived 
changing period) (step S205). Specifically, the rhythm 

35 estimating section 213 derives a standard deviation of 
samples in the sound pressure data after the application 
of the inverse FFT process and compares it with a pre- 
determined ttnreshold value. If the standard deviation is 
smaller than tiie threshold value, meaning tiiat there is 

40 no sound pressure protruding portion in the sound pres- 
sure data, the rhythm estimating section 213 judges the 
derived changing period to be incorrect. In this event, 
tiie sequence returns to step S204 where tiie rhythm 
estimating section 213 changes the period of the auxil- 

45 iary data and carries out the inverse FFT process (No at 
step S206). On the other hand, if the derived changing 
period is judged to be correct, the rhyttim estimating 
section 213 extracts a rhythm component based on the 
verified data (step S207). Fig. 5 at (i) shows a waveform 

so of the extracted rhythm component (con^esponding to 
the frequency band B in Fig. 5). 
[0073] The rhythm estimating section 213 esti- 
mates or presumes a subsequent rhythm component 
and outputs it for a predetermined time (step S208). Fig. 

55 5 at (j) shows the presumed rhythm component After a 
lapse of the predetermined time, the sequence returns 
to step S201 (Yes at step S209). 
[0074] The foregoing predetermined time may be 
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up to the end of the music or may correspond to only 
several frames. 

[0075] By executing the foregoing operation 
sequence, the rhythm component can be extracted 
quickly and accurately from the musical data so that the 5 
timing of changing the character figure can be approxi- 
mated to the rhythm beaten in the reproduced music. 
Thus, the game player can visually grasp an image of 
the reproduced music. 

[0076] As described above, in the foregoing music io 
reproducing system 2. when the rhythm component of 
the reproduced music does not agree with the favorite 
rhythm component of the displayed character, the figure 
expressing ability of the displayed character is lowered. 
Thus, as shown in Fig. 6, the motion, expression and is 
background of the character scarcely change while 
music is reproduced. 

[0077] In contrast, as shown in Fig. 7, when music 
matching with the favorite rhythm component of the 
character is reproduced, the motion of the character 20 
becomes active and the expression thereof also 
becomes affluent. 

[0078] Further, as shown in Fig. 8, when music 
matching with the favorite rfiythm component is repeat- 
edly reproduced, the figure expressing ability of the 2S 
character is cumulatively enhanced so that the motion 
of the character becomes further active and the expres- 
sion thereof also becomes further affluent. In addition, 
the background image is also largely changed accord- 
ing to the magnitude of the figure expressing ability, and 30 
further, the effect image is displayed every time the fig- 
ure expressing ability reaches the reference value. 
[0079] In this fashion, the game player can objec- 
tively and accurately seize, by means of CG images, the 
progress of the display figure of the character changing 35 
depending on music the game player listens to. so that 
attraction can be enhanced. 

[0080] As desCTibed above, since the display figure 
of the character can be changed in real time according 
to music, i.e. general entertainment, and further accord- 
ing to the accurate rhythm component thereof, the 
music reproducing system can be enjoyed by a wide 
range of game players. 

[0081] In the foregoing preferred embodiment, for 
brevity of description, each character Is allocated only 45 
one favorite rhythm component (I.e. one of the first to 
third rhythm bands), and it is only judged whether a 
rhythm component extracted from musical data agrees 
with the allocated favorite rhythm component. On the 
other hand, it may also be arranged that different so 
weighting data are set to a plurality of rhythm bands and 
that the figure expressing ability is increased more 
remarkably as the extracted rhythm component 
approaches the favorite rhythm component, while the 
figure expressing ability is lowered more remarkably as 55 
the extracted rhythm component goes away from the 
favorite rhythm component. It may also be arranged that 
when the extracted rhythm component does not agree 



14 

with the favorite rhythm component, the figure express- 
ing ability is temporarily replaced with another parame- 
ter, for example, a parameter which suppresses the 
changing degree of the display figure, instead of lower- 
ing the figure expressing ability. 
[0082] It may also be arranged that the figure 
expressing ability is divided into, for example, one for 
character motion display, one for character expression 
display, one for background display and one for item dis- 
play, and that when a certain rhythm component is 
extracted, only the motion of a character is changed. 
With this arrangement, the display figure of the charac- 
ter based on nrtuslcal data can be more diversified. 
[0083] In the foregoing preferred embodiment, the 
display figure of the character and its background are 
changed based on the control information containing a 
parameter of the figure expressing ability. On the other 
hand, it may also be arranged that the control informa- 
tion includes only a difference or deviation between the 
extracted rhythm component and the favorite rhythm 
component, instead of the figure expressing ability. 
When applied to competition games, there may be a 
case wherein exclusion of the cumulative change of the 
figure expressing ability is desirable. 
[0084] In the foregoing preferred embodiment, 
musical data is stored in the general music CD other 
than the CD-ROM 44 storing the game program, and 
the music CD is loaded into the video game apparatus 1 
while the music reproducing system 2 is constituted in 
the video game apparatus 1 . On the other hand, it may 
also be arranged that the musical data and the game 
program are stored in the same storage medium. 
[0085] As seen from the foregoing description, 
according to the present invention, a rhythm component 
is accurately analyzed based on desired musical data 
so that, for example, in a competition game using a 
result of the rhythm analysis, a competition result can 
be more objectified. 

[0086] Further, since a display figure of a displayed 
character is changed in real time and accurately accord- 
ing to a rhythm component, an image of music can be 
correctly visualized. 

[0087] While the present invention has been 
described in terms of the preferred embodiment and the 
modifications thereof the invention is not to be limited 
thereto, but can be embodied in various ways without 
departing from the principle of the invention as defined 
in the appended claims. 

Claims 

1 . A music reproducing system wherein a character is 
allocated in advance a favorite rhythm component 
and a display figure of the character on a screen is 
changed based on musical data, said system com- 
prising: 

frequency band identifying means for produc- 
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ing sound pressure data for each of a plurality 
of frequency bands from said musical data and 
for identifying, based on the sound pressure 
data for the plurality of frequency bands, one of 
the plurality of frequency bands which beats a s 
rhythm most: 

rhythm estimating means for deriving a chang- 
ing period of a sound pressure component 
from the sound pressure data for the identified 
frequency band and for estimating a rhythm io 
component of the sound pressure data for the 
identified frequency band based on the derived 
changing period: and 

control means for changing the display figure of 
the character according to a matching degree is 
between the estimated rtiythm component and 
the favorite rhythm component. 

The music reproducing system according to claim 
1 . wherein said musical data is stored in a portable 20 
storage medium and said frequency band identify- 
ing means is activated when said musical data is 
read into said music reproducing system from said 
portable storage medium. 

25 

The music reproducing system according to claim 
1, wherein said rhythm estimating means derives 
said changing period by applying an FFT process to 
the sound pressure data for said identified fre- 
quency band, superimposes auxiliary data having a 30 
period integer times said changing period onto the 6. 
sound pressure data for said identified frequency 
band so as to suppress components other than the 
rhythm component, and applies an inverse FFT 
process to the sound pressure data superimposed 35 
with said auxiliary data so as to estimate said 
rhythm component. 

The music reproducing system according to claim 
3, wherein said rhythm estimating means verifies 4o 
whether said changing period is correct and. when 
said changing period Is verified to be correct, said 
rhythm estimating means outputs said rhythm oonfv 
ponent for a predetermined time. 8. 

45 

A music reproducing system connected to a display 
unit and provided with music inputting means and 
music reproducing means, said system comprising: 9. 

first storing means for storing a favorite rhythm so 
component for each of characters; 
second storing means for storing a figure 
expressing ability for each of said characters, 
said figure expressing ability used for express- 
ing a display figure of the corresponding char- ss 
acter to be displayed on a screen of said 
display unit; 

frequency band identifying means for produc- 



ing sound pressure data for each of a plurality 
of frequency bands from musical data inputted 
via said music inputting means, and for identify- 
ing, based on the sound pressure data for the 
plurality of frequency bands, one of the plurality 
of frequency bands which beats a rhythm most: 
rhythm estimating means lor deriving a chang- 
ing period of a sound pressure component 
from the sound pressure data for the identified 
frequency band and for estimating a rhythm 
component of the sound pressure data for the 
identified frequency band based on the derived 
changing period; and 

control means for causing one of said charac- 
ters to be displayed on the saeen of said dis* 
play unit, reading said favorite rhythm 
component of said one of said characters from 
said first storing means, updating the figure 
expressing ability of said one of said characters 
stored in said second storing means according 
to a matching degree between said favorite 
rhythm component read from said first storing 
means and said rhythm component estirriated 
by said rhythm estimating means, and chang- 
ing the display figure of said displayed charac- 
ter based on the updated figure expressing 
ability to the accompaniment of the musical 
data reproduced by said music reproducing 
means. 

The music reproducing system according to claim 
5, wherein said control means updates the figure 
expressing ability of said displayed character so as 
to be cumulatively increased every time musical 
data containing said favorite rhythm component is 
reproduced. 

The music reproducing system according to claim 
5. further comprising background image control 
means for changing a background image of said 
displayed character according to said figure 
expressing ability. 

The music reproducing system according to claim 
5. wherein said second storing means is an 
exchangeable nonvolatile storage medium. 

A rhythm analyzing method to be implemented in 
an apparatus which displays a character allocated 
in advance a favorite rhythm component on a 
screen based on a rhythm component of musical 
data reproduced, said method comprising the steps 
of: 

producing sound pressure data for each of a 
plurality of frequency bands from said musical 
data, and identifying, based on the sound pres- 
sure data for the plurality of frequency bands. 
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one of the plurality of frequency bands which 
beats a rhythm most; 

deriving a changing period of a sound pressure 
component from the sound pressure data for 
the identified frequency band, and estimating a s 
rhythm component of the sound pressure data 
for the identified frequency band based on the 
derived changing period; and 
reflecting a matching degree between the esti- 
mated rhythm component and the favorite io 
rhythm component onto control information for 
changing a display figure of the displayed char- 
acter. 

10. The rhythm analyzing method according to claim 9. is 
wherein the step of deriving said changing period 
and estimating said rhythm component comprises 
deriving said changing period by applying an FFT 
process to the sound pressure data for said identi- 
fied frequency band, superimposing auxiliary data 20 
having a period integer times said changing period 
onto the sound pressure data for said identified fre- 
quency band so as to suppress components other 
than the rhythm component, applying an inverse 
FFT process to the sound pressure data superim- 25 
posed with said auxiliary data and performing a ver- 
ification process for the sound pressure data 
applied with said inverse FFT process so as to esti- 
mate said rhythm component. 

30 

11. The rhythm analyzing method according to daim 9, 
further conprising outputting said estimated rhythm 
component for a predetermined time. 



13. The storage medium according to claim 12. 
wherein said program codes cause an operating 
system of said computer or other program codes to 
execute a part of said steps. 

14. The storage medium according to claim 12, 
wherein said storage medium also stores said 
musical data and is separable from said computer. 

15. A storage medium storing program codes readable 
by a computer which displays a character allocated 
in advance a favorite rhythm component on a 
screen based on a figure expressing ability peculiar 
to the character and a rhythm component of musi- 
cal data reproduced, said program codes causing 
said computer to execute the steps of: 

producing sound pressure data for each of a 
plurality of frequency bands from said musical 
data, and identifying, based on the sourxJ pres- 
sure data for the plurality of frequency bands, 
one of the plurality of frequency bands which 
beats a rhythm most; 

deriving a changing period of a sound pressure 
component from the sound pressure data for 
the identified frequency band, and estimating a 
rhythm component of the sound pressure data 
for the identified frequency band based on the 
derived changing period; and 
updating said figure expressing ability accord- 
ing to a matching degree between the esti- 
mated rhythm conponent and the favorite 
rhythm component. 



12. A storage medium storing program codes readable 3S 
by a computer which displays a character allocated 
in advance a favorite rhythm component on a 
screen based on a rhythm component of musical 
data, said program codes causing said computer to 
execute the steps of: 40 

producing sound pressure data for each of a 
plurality of frequency bands from said musical 
data, and identifying, based on the sound pres- 
sure data for the plurality of frequency bands. 4S 
one of the plurality of frequency bands which 
beats a rhythm most; 

deriving a changing period of a sound pressure 
conponent from the sound pressure data for 
the identified frequency band, and estimating a so 
rhythm component of the sound pressure data 
for the identified frequency band based on the 
derived changing period; and 
reflecting a matching degree between the esti- 
mated rhythm component and the favorite ss 
rhythm component onto control information for 
changing a display figure of the displayed char- 
acter. 



16. The storage medium according to claim 15, 
wherein said program codes cause said computer 
to execute a step of changing a background image 
of said character according to said figure express- 
ing ability. 

17. The storage medium according to claim 15. 
wherein said program codes cause an operating 
system of said computer or other program codes to 
execute a part of said steps. 

18. The storage medium according to claim 15, 
wherein said storage medium also stores said 
musical data and is separable from said computer. 
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(54) Music reproducing system, rhythm analyzing method and storage medium 



(57) In a music reproducing system, each of charac- 
ters is allocated in advance a favorite rhythm compo- 
nent and a peculiar figure expressing ability. A sound 
pressure data producing section produces sound pres- 
sure data for each of a plurality of frequency bands from 
musical data. A frequency band identifying section iden- 
tifies the frequency band which beats a rhythm most. A 
rhythm estimating section derives a changing period of 
a sound pressure component in the sound pressure 
data at the identified frequency band and estimates a 
rhythm component based on the derived changing 
period. A character managing section cumulatively 
changes a figure expressing ability of a displayed char- 
acter according to the matching degree between the 
estimated rhythm component and a favorite rhythm 
component of the displayed character. A display control 
section changes a display figure of the displayed char- 
acter according to the figure expressing ability thereof 
while musical data is reproduced. 
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(54) Music reproducing system, rhythm analyzing method and storage medium 



(57) In a music reproducing system, each of char- 
acters is allocated in advance a favorite rhythm compo- 
nent and a peculiar figure expressing ability. A sound 
pressure data producing section produces sound pres- 
sure data for each of a plurality of frequency bands from 
musical data. A frequency band identifying section iden- 
tifies the frequency band which beats a rhythm most. A 
rhythm estimating section derives a changing period of 
a sound pressure component in the sound pressure da- 
ta at the identified frequency band and estimates a 
rhythm component based on the derived changing pe- 
riod, A character managing section cumulatively chang- 
es a figure expressing ability of a displayed character 
according to the matching degree between the estimat- 
ed rhythm component and a favorite rhythm component 
of the displayed character. A display control section 
changes a display figure of the displayed character ac- 
cording to the figure expressing ability thereof while mu- 
sical data is reproduced. 
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